{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c80c3c75-9bbe-4107-8ebd-96301db983f3",
   "metadata": {},
   "source": [
    "(parallel-workflows)=\n",
    "# Running a workflow with multiple functions in parallel\n",
    "\n",
    "This example illustrates a workflow that triggers multiple functions in parallel. The \"parent\" functions run in parallel, and each parent function encapsulates multiple functions that run serially.\n",
    "\n",
    "Use this type of workflow for anything that needs to be done in parallel. For example, data processing on multiple data sources and then combining them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7046cb2b-8ef3-4f6e-94a2-ddeeed3f81c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2025-05-05 11:33:47,178 [info] Server and client versions are not the same but compatible: {'parsed_server_version': Version(major=1, minor=8, patch=0, prerelease='rc53', build=None), 'parsed_client_version': Version(major=1, minor=6, patch=4, prerelease=None, build=None)}\n"
     ]
    }
   ],
   "source": [
    "import mlrun"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e672313-504a-4f95-aac5-5df271f020ad",
   "metadata": {},
   "source": [
    "## Create the project and the function\n",
    "\n",
    "Create a function that prints to the logger. This is the function that will be encapsulated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fe217992-a316-4054-b0ca-8536d06e1385",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2025-05-05 11:33:47,240 [info] Project loaded successfully: {'project_name': 'parallel-workflow'}\n"
     ]
    }
   ],
   "source": [
    "project = mlrun.get_or_create_project(\"parallel-workflow\", \"./\", user_project=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "04dad68c-60da-47a2-a6bd-4a52ae6301aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting func.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile func.py\n",
    "\n",
    "import mlrun\n",
    "\n",
    "# Function that prints to the logger i*100 for i that is 0-20.\n",
    "def func(context):\n",
    "    for i in range(5):\n",
    "        context.logger.info(str(i)*100)\n",
    "        \n",
    "    return 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c956ec8a-bec3-4374-9a14-b827b78ad1e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f1eefea8400>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set the function in the project\n",
    "project.set_function(\n",
    "    name=\"func\", func=\"func.py\", image=\"mlrun/mlrun\", kind=\"job\", handler=\"func\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffbef104-9098-4889-8d39-b26fd3c14bc2",
   "metadata": {},
   "source": [
    "## Run the function \n",
    "\n",
    "Run the function to observe the output: 4 functions running serially."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2c4cce38-e793-4627-b87e-7832a9927636",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2025-05-05 11:33:47,316 [info] Storing function: {'name': 'func-func', 'uid': '1df442b770724582bf0855db667322d0', 'db': 'http://mlrun-api:8080'}\n",
      "> 2025-05-05 11:33:47,635 [info] Job is running in the background, pod: func-func-hh44l\n",
      "> 2025-05-05 11:33:51,144 [info] Server and client versions are not the same but compatible: {'parsed_server_version': Version(major=1, minor=8, patch=0, prerelease='rc53', build=None), 'parsed_client_version': Version(major=1, minor=6, patch=4, prerelease=None, build=None)}\n",
      "> 2025-05-05 11:33:51,467 [info] 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\n",
      "> 2025-05-05 11:33:51,467 [info] 1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111\n",
      "> 2025-05-05 11:33:51,467 [info] 2222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222\n",
      "> 2025-05-05 11:33:51,467 [info] 3333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333\n",
      "> 2025-05-05 11:33:51,467 [info] 4444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444\n",
      "> 2025-05-05 11:33:51,685 [info] To track results use the CLI: {'info_cmd': 'mlrun get run 1df442b770724582bf0855db667322d0 -p parallel-workflow-jill', 'logs_cmd': 'mlrun logs 1df442b770724582bf0855db667322d0 -p parallel-workflow-jill'}\n",
      "> 2025-05-05 11:33:51,685 [info] Or click for UI: {'ui_url': 'https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/1df442b770724582bf0855db667322d0/overview'}\n",
      "> 2025-05-05 11:33:51,685 [info] Run execution finished: {'status': 'completed', 'name': 'func-func'}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>parallel-workflow-jill</td>\n",
       "      <td><div title=\"1df442b770724582bf0855db667322d0\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/1df442b770724582bf0855db667322d0/overview\" target=\"_blank\" >...667322d0</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>May 05 11:33:51</td>\n",
       "      <td>completed</td>\n",
       "      <td>func-func</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=Jill</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=Jill</div><div class=\"dictlist\">mlrun/client_version=1.6.4</div><div class=\"dictlist\">mlrun/client_python_version=3.9.18</div><div class=\"dictlist\">host=func-func-hh44l</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resulte666e3e9-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resulte666e3e9-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resulte666e3e9\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resulte666e3e9-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/1df442b770724582bf0855db667322d0/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2025-05-05 11:33:56,903 [info] Run execution finished: {'status': 'completed', 'name': 'func-func'}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.model.RunObject at 0x7f1eefec9580>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.run_function(\"func\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c297206-b693-4e0d-a1de-89a6db2912cc",
   "metadata": {},
   "source": [
    "## Create the workflow\n",
    "\n",
    "The `pipe_num` determines how many pipelines run in parallel in the workflow. Each pipeline creates 20 jobs that run one by one.\n",
    "\n",
    "In this example, `pipe_num=4`, 4 pipelines in parallel. Each pipeline runs 5 jobs, \n",
    "giving a total of 4*5=20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "402c9cbc-224d-4de2-82de-939adfa2e53b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting workflow.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile workflow.py\n",
    "\n",
    "from kfp import dsl\n",
    "from mlrun.platforms import auto_mount\n",
    "import os\n",
    "import sys\n",
    "import mlrun\n",
    "\n",
    "pipe_num = 4\n",
    "def kfpipeline():\n",
    "    with dsl.ParallelFor([i for i in range(pipe_num)]) as item:\n",
    "        step_1 = mlrun.run_function('func')\n",
    "        step_2 = mlrun.run_function('func').after(step_1)\n",
    "        step_3 = mlrun.run_function('func').after(step_2)\n",
    "        step_4 = mlrun.run_function('func').after(step_3)\n",
    "        step_5 = mlrun.run_function('func').after(step_4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d41b9e0",
   "metadata": {},
   "source": [
    "(parallelism-4881)=\n",
    "```{admonition} Note\n",
    "The \"parallelism\" parameter of `dsl.ParallelFor` is not supported (see the [Kubeflow documentation](https://www.kubeflow.org/docs/components/pipelines/user-guides/core-functions/control-flow/#dslparallelfor)). You can use the `dsl.SubGraph(parallelism=)` component to limit parallel executions, however the workflow graph does not display in MLRun UI until the workflow completes. For example:\n",
    "```python\n",
    "pipe_num = 4\n",
    "def kfpipeline():\n",
    "    with dsl.SubGraph(parallelism=1)\"\n",
    "        with dsl.ParallelFor([i for i in range(pipe_num)]) as item:\n",
    "             step_1 = mlrun.run_function('func')\n",
    "             step_2 = mlrun.run_function('func').after(step_1)\n",
    "             step_3 = mlrun.run_function('func').after(step_2)\n",
    "             step_4 = mlrun.run_function('func').after(step_3)\n",
    "             step_5 = mlrun.run_function('func').after(step_4)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "eb4f9126-bab3-4da4-8439-5890ca1e2a92",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>Pipeline running (id=9e4f313d-fe67-4a5c-a3fc-abb3ac40e9ca), <a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor-workflows/workflow/9e4f313d-fe67-4a5c-a3fc-abb3ac40e9ca\" target=\"_blank\"><b>click here</b></a> to view the details in MLRun UI</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: kfp Pages: 1 -->\n",
       "<svg width=\"674pt\" height=\"476pt\"\n",
       " viewBox=\"0.00 0.00 674.39 476.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 472)\">\n",
       "<title>kfp</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-472 670.39,-472 670.39,4 -4,4\"/>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;1010339658 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;1010339658</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"76.69\" cy=\"-234\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"76.69\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;2</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;1027117277 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;1027117277</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"76.69\" cy=\"-162\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"76.69\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;3</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;1010339658&#45;&gt;kfpipeline&#45;k8jsf&#45;1027117277 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;1010339658&#45;&gt;kfpipeline&#45;k8jsf&#45;1027117277</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M76.69,-215.7C76.69,-207.98 76.69,-198.71 76.69,-190.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"80.19,-190.1 76.69,-180.1 73.19,-190.1 80.19,-190.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;909673944 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;909673944</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"76.69\" cy=\"-90\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"76.69\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;4</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;1027117277&#45;&gt;kfpipeline&#45;k8jsf&#45;909673944 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;1027117277&#45;&gt;kfpipeline&#45;k8jsf&#45;909673944</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M76.69,-143.7C76.69,-135.98 76.69,-126.71 76.69,-118.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"80.19,-118.1 76.69,-108.1 73.19,-118.1 80.19,-118.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;926451563 -->\n",
       "<g id=\"node25\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;926451563</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"76.69\" cy=\"-18\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"76.69\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;5</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;909673944&#45;&gt;kfpipeline&#45;k8jsf&#45;926451563 -->\n",
       "<g id=\"edge24\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;909673944&#45;&gt;kfpipeline&#45;k8jsf&#45;926451563</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M76.69,-71.7C76.69,-63.98 76.69,-54.71 76.69,-46.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"80.19,-46.1 76.69,-36.1 73.19,-46.1 80.19,-46.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;1157894533 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;1157894533</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"589.69\" cy=\"-306\" rx=\"30.59\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"589.69\" y=\"-302.3\" font-family=\"Times,serif\" font-size=\"14.00\">func</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;610600318 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;610600318</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"589.69\" cy=\"-234\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"589.69\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;2</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;1157894533&#45;&gt;kfpipeline&#45;k8jsf&#45;610600318 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;1157894533&#45;&gt;kfpipeline&#45;k8jsf&#45;610600318</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M589.69,-287.7C589.69,-279.98 589.69,-270.71 589.69,-262.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"593.19,-262.1 589.69,-252.1 586.19,-262.1 593.19,-262.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;627377937 -->\n",
       "<g id=\"node22\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;627377937</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"589.69\" cy=\"-162\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"589.69\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;3</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;610600318&#45;&gt;kfpipeline&#45;k8jsf&#45;627377937 -->\n",
       "<g id=\"edge19\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;610600318&#45;&gt;kfpipeline&#45;k8jsf&#45;627377937</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M589.69,-215.7C589.69,-207.98 589.69,-198.71 589.69,-190.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"593.19,-190.1 589.69,-180.1 586.19,-190.1 593.19,-190.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;1470552911 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;1470552911</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"76.69\" cy=\"-378\" rx=\"76.89\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"76.69\" y=\"-374.3\" font-family=\"Times,serif\" font-size=\"14.00\">for&#45;loop&#45;2(0:0)</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;4172880825 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;4172880825</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"76.69\" cy=\"-306\" rx=\"30.59\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"76.69\" y=\"-302.3\" font-family=\"Times,serif\" font-size=\"14.00\">func</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;1470552911&#45;&gt;kfpipeline&#45;k8jsf&#45;4172880825 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;1470552911&#45;&gt;kfpipeline&#45;k8jsf&#45;4172880825</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M76.69,-359.7C76.69,-351.98 76.69,-342.71 76.69,-334.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"80.19,-334.1 76.69,-324.1 73.19,-334.1 80.19,-334.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;4172880825&#45;&gt;kfpipeline&#45;k8jsf&#45;1010339658 -->\n",
       "<g id=\"edge18\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;4172880825&#45;&gt;kfpipeline&#45;k8jsf&#45;1010339658</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M76.69,-287.7C76.69,-279.98 76.69,-270.71 76.69,-262.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"80.19,-262.1 76.69,-252.1 73.19,-262.1 80.19,-262.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;2105347502 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;2105347502</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"247.69\" cy=\"-234\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"247.69\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;2</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;2122125121 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;2122125121</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"247.69\" cy=\"-162\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"247.69\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;3</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;2105347502&#45;&gt;kfpipeline&#45;k8jsf&#45;2122125121 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;2105347502&#45;&gt;kfpipeline&#45;k8jsf&#45;2122125121</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M247.69,-215.7C247.69,-207.98 247.69,-198.71 247.69,-190.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"251.19,-190.1 247.69,-180.1 244.19,-190.1 251.19,-190.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;2138902740 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;2138902740</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"247.69\" cy=\"-90\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"247.69\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;4</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;2122125121&#45;&gt;kfpipeline&#45;k8jsf&#45;2138902740 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;2122125121&#45;&gt;kfpipeline&#45;k8jsf&#45;2138902740</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M247.69,-143.7C247.69,-135.98 247.69,-126.71 247.69,-118.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"251.19,-118.1 247.69,-108.1 244.19,-118.1 251.19,-118.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;2155680359 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;2155680359</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"247.69\" cy=\"-18\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"247.69\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;5</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;2138902740&#45;&gt;kfpipeline&#45;k8jsf&#45;2155680359 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;2138902740&#45;&gt;kfpipeline&#45;k8jsf&#45;2155680359</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M247.69,-71.7C247.69,-63.98 247.69,-54.71 247.69,-46.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"251.19,-46.1 247.69,-36.1 244.19,-46.1 251.19,-46.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;2787249875 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;2787249875</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"247.69\" cy=\"-378\" rx=\"76.89\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"247.69\" y=\"-374.3\" font-family=\"Times,serif\" font-size=\"14.00\">for&#45;loop&#45;2(1:1)</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3800325845 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3800325845</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"247.69\" cy=\"-306\" rx=\"30.59\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"247.69\" y=\"-302.3\" font-family=\"Times,serif\" font-size=\"14.00\">func</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;2787249875&#45;&gt;kfpipeline&#45;k8jsf&#45;3800325845 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;2787249875&#45;&gt;kfpipeline&#45;k8jsf&#45;3800325845</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M247.69,-359.7C247.69,-351.98 247.69,-342.71 247.69,-334.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"251.19,-334.1 247.69,-324.1 244.19,-334.1 251.19,-334.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3800325845&#45;&gt;kfpipeline&#45;k8jsf&#45;2105347502 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3800325845&#45;&gt;kfpipeline&#45;k8jsf&#45;2105347502</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M247.69,-287.7C247.69,-279.98 247.69,-270.71 247.69,-262.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"251.19,-262.1 247.69,-252.1 244.19,-262.1 251.19,-262.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3215922176 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3215922176</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"418.69\" cy=\"-90\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"418.69\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;4</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3232699795 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3232699795</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"418.69\" cy=\"-18\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"418.69\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;5</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3215922176&#45;&gt;kfpipeline&#45;k8jsf&#45;3232699795 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3215922176&#45;&gt;kfpipeline&#45;k8jsf&#45;3232699795</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M418.69,-71.7C418.69,-63.98 418.69,-54.71 418.69,-46.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"422.19,-46.1 418.69,-36.1 415.19,-46.1 422.19,-46.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3316587890 -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3316587890</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"418.69\" cy=\"-234\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"418.69\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;2</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3333365509 -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3333365509</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"418.69\" cy=\"-162\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"418.69\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;3</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3316587890&#45;&gt;kfpipeline&#45;k8jsf&#45;3333365509 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3316587890&#45;&gt;kfpipeline&#45;k8jsf&#45;3333365509</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M418.69,-215.7C418.69,-207.98 418.69,-198.71 418.69,-190.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"422.19,-190.1 418.69,-180.1 415.19,-190.1 422.19,-190.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3333365509&#45;&gt;kfpipeline&#45;k8jsf&#45;3215922176 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3333365509&#45;&gt;kfpipeline&#45;k8jsf&#45;3215922176</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M418.69,-143.7C418.69,-135.98 418.69,-126.71 418.69,-118.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"422.19,-118.1 418.69,-108.1 415.19,-118.1 422.19,-118.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3990359006 -->\n",
       "<g id=\"node18\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3990359006</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"332.69\" cy=\"-450\" rx=\"55.49\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"332.69\" y=\"-446.3\" font-family=\"Times,serif\" font-size=\"14.00\">for&#45;loop&#45;2</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3990359006&#45;&gt;kfpipeline&#45;k8jsf&#45;1470552911 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3990359006&#45;&gt;kfpipeline&#45;k8jsf&#45;1470552911</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M291.85,-437.83C249.41,-426.23 182.63,-407.97 134.73,-394.87\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"135.38,-391.42 124.81,-392.16 133.53,-398.17 135.38,-391.42\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3990359006&#45;&gt;kfpipeline&#45;k8jsf&#45;2787249875 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3990359006&#45;&gt;kfpipeline&#45;k8jsf&#45;2787249875</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M313.41,-433.12C302.24,-423.92 287.98,-412.18 275.6,-401.98\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"277.74,-399.21 267.79,-395.55 273.29,-404.61 277.74,-399.21\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;820040535 -->\n",
       "<g id=\"node19\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;820040535</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"418.69\" cy=\"-378\" rx=\"76.89\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"418.69\" y=\"-374.3\" font-family=\"Times,serif\" font-size=\"14.00\">for&#45;loop&#45;2(2:2)</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3990359006&#45;&gt;kfpipeline&#45;k8jsf&#45;820040535 -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3990359006&#45;&gt;kfpipeline&#45;k8jsf&#45;820040535</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M352.21,-433.12C363.51,-423.92 377.93,-412.18 390.46,-401.98\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"392.81,-404.58 398.36,-395.55 388.39,-399.15 392.81,-404.58\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;848880515 -->\n",
       "<g id=\"node20\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;848880515</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"589.69\" cy=\"-378\" rx=\"76.89\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"589.69\" y=\"-374.3\" font-family=\"Times,serif\" font-size=\"14.00\">for&#45;loop&#45;2(3:3)</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;3990359006&#45;&gt;kfpipeline&#45;k8jsf&#45;848880515 -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;3990359006&#45;&gt;kfpipeline&#45;k8jsf&#45;848880515</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M373.7,-437.83C416.39,-426.2 483.62,-407.89 531.73,-394.79\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"532.66,-398.16 541.39,-392.16 530.82,-391.41 532.66,-398.16\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;4031374161 -->\n",
       "<g id=\"node21\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;4031374161</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"418.69\" cy=\"-306\" rx=\"30.59\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"418.69\" y=\"-302.3\" font-family=\"Times,serif\" font-size=\"14.00\">func</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;820040535&#45;&gt;kfpipeline&#45;k8jsf&#45;4031374161 -->\n",
       "<g id=\"edge22\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;820040535&#45;&gt;kfpipeline&#45;k8jsf&#45;4031374161</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M418.69,-359.7C418.69,-351.98 418.69,-342.71 418.69,-334.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"422.19,-334.1 418.69,-324.1 415.19,-334.1 422.19,-334.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;848880515&#45;&gt;kfpipeline&#45;k8jsf&#45;1157894533 -->\n",
       "<g id=\"edge23\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;848880515&#45;&gt;kfpipeline&#45;k8jsf&#45;1157894533</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M589.69,-359.7C589.69,-351.98 589.69,-342.71 589.69,-334.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"593.19,-334.1 589.69,-324.1 586.19,-334.1 593.19,-334.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;4031374161&#45;&gt;kfpipeline&#45;k8jsf&#45;3316587890 -->\n",
       "<g id=\"edge17\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;4031374161&#45;&gt;kfpipeline&#45;k8jsf&#45;3316587890</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M418.69,-287.7C418.69,-279.98 418.69,-270.71 418.69,-262.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"422.19,-262.1 418.69,-252.1 415.19,-262.1 422.19,-262.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;644155556 -->\n",
       "<g id=\"node23\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;644155556</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"589.69\" cy=\"-90\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"589.69\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;4</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;627377937&#45;&gt;kfpipeline&#45;k8jsf&#45;644155556 -->\n",
       "<g id=\"edge20\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;627377937&#45;&gt;kfpipeline&#45;k8jsf&#45;644155556</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M589.69,-143.7C589.69,-135.98 589.69,-126.71 589.69,-118.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"593.19,-118.1 589.69,-108.1 586.19,-118.1 593.19,-118.1\"/>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;660933175 -->\n",
       "<g id=\"node24\" class=\"node\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;660933175</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"589.69\" cy=\"-18\" rx=\"39.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"589.69\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">func&#45;5</text>\n",
       "</g>\n",
       "<!-- kfpipeline&#45;k8jsf&#45;644155556&#45;&gt;kfpipeline&#45;k8jsf&#45;660933175 -->\n",
       "<g id=\"edge21\" class=\"edge\">\n",
       "<title>kfpipeline&#45;k8jsf&#45;644155556&#45;&gt;kfpipeline&#45;k8jsf&#45;660933175</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M589.69,-71.7C589.69,-63.98 589.69,-54.71 589.69,-46.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"593.19,-46.1 589.69,-36.1 586.19,-46.1 593.19,-46.1\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f1eefe32820>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h2>Run Results</h2><h3>[info] Workflow 9e4f313d-fe67-4a5c-a3fc-abb3ac40e9ca finished, state=Succeeded</h3><br>click the hyper links below to see detailed results<br><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>uid</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td><div title=\"72026455b4a8413089f244bb4320af6c\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/72026455b4a8413089f244bb4320af6c/overview\" target=\"_blank\" >...4320af6c</a></div></td>\n",
       "      <td>May 05 11:36:09</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"f61510a2a7b046ecaf007a592fd6322f\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/f61510a2a7b046ecaf007a592fd6322f/overview\" target=\"_blank\" >...2fd6322f</a></div></td>\n",
       "      <td>May 05 11:36:09</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"dab0c957854842459cc5100878b743a6\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/dab0c957854842459cc5100878b743a6/overview\" target=\"_blank\" >...78b743a6</a></div></td>\n",
       "      <td>May 05 11:36:09</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"72b12902800b4d33b423de3655cc9e29\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/72b12902800b4d33b423de3655cc9e29/overview\" target=\"_blank\" >...55cc9e29</a></div></td>\n",
       "      <td>May 05 11:36:09</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"16591bd73fe148d3a0e8adce909c5ba0\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/16591bd73fe148d3a0e8adce909c5ba0/overview\" target=\"_blank\" >...909c5ba0</a></div></td>\n",
       "      <td>May 05 11:35:38</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"024d00628a4f41c891390febc9af0a39\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/024d00628a4f41c891390febc9af0a39/overview\" target=\"_blank\" >...c9af0a39</a></div></td>\n",
       "      <td>May 05 11:35:38</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"9f4b53872cf344178be9929af00d7542\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/9f4b53872cf344178be9929af00d7542/overview\" target=\"_blank\" >...f00d7542</a></div></td>\n",
       "      <td>May 05 11:35:38</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"c771d7d21d47408181d1e24a44be7b76\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/c771d7d21d47408181d1e24a44be7b76/overview\" target=\"_blank\" >...44be7b76</a></div></td>\n",
       "      <td>May 05 11:35:37</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"119f8d0a93c14ab9a2e1fefd38d697a4\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/119f8d0a93c14ab9a2e1fefd38d697a4/overview\" target=\"_blank\" >...38d697a4</a></div></td>\n",
       "      <td>May 05 11:35:07</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"1a6a94c8652f4d2c8505895cf29aaeb4\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/1a6a94c8652f4d2c8505895cf29aaeb4/overview\" target=\"_blank\" >...f29aaeb4</a></div></td>\n",
       "      <td>May 05 11:35:07</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"9419fedbcf7b4ca199002cfc573d13c1\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/9419fedbcf7b4ca199002cfc573d13c1/overview\" target=\"_blank\" >...573d13c1</a></div></td>\n",
       "      <td>May 05 11:35:07</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"4d1e41ddc7024e84beb1d294eb497570\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/4d1e41ddc7024e84beb1d294eb497570/overview\" target=\"_blank\" >...eb497570</a></div></td>\n",
       "      <td>May 05 11:35:07</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"d1d6c98d7b5840f0a7f66b8aa9c42e87\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/d1d6c98d7b5840f0a7f66b8aa9c42e87/overview\" target=\"_blank\" >...a9c42e87</a></div></td>\n",
       "      <td>May 05 11:34:40</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"8f0cec07240d4d5798e181b105d77987\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/8f0cec07240d4d5798e181b105d77987/overview\" target=\"_blank\" >...05d77987</a></div></td>\n",
       "      <td>May 05 11:34:40</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"79cb8901c9f24b9bb5c9ddb1f64d147a\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/79cb8901c9f24b9bb5c9ddb1f64d147a/overview\" target=\"_blank\" >...f64d147a</a></div></td>\n",
       "      <td>May 05 11:34:40</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"eb4abbcc835f48619ee2454a0fa9c712\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/eb4abbcc835f48619ee2454a0fa9c712/overview\" target=\"_blank\" >...0fa9c712</a></div></td>\n",
       "      <td>May 05 11:34:40</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"3fde59a45d76401aabb9ed71f14f03e4\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/3fde59a45d76401aabb9ed71f14f03e4/overview\" target=\"_blank\" >...f14f03e4</a></div></td>\n",
       "      <td>May 05 11:34:12</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"5d6cbdfc91de4f0f8951ce9e53abd3f5\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/5d6cbdfc91de4f0f8951ce9e53abd3f5/overview\" target=\"_blank\" >...53abd3f5</a></div></td>\n",
       "      <td>May 05 11:34:11</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"b7aa905355a84ee0a5b4484e32574335\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/b7aa905355a84ee0a5b4484e32574335/overview\" target=\"_blank\" >...32574335</a></div></td>\n",
       "      <td>May 05 11:34:11</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"8e070c3525af4e33a64300029f3e07f8\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il.iguazio-cd0.com/mlprojects/parallel-workflow-jill/jobs/monitor/8e070c3525af4e33a64300029f3e07f8/overview\" target=\"_blank\" >...9f3e07f8</a></div></td>\n",
       "      <td>May 05 11:34:11</td>\n",
       "      <td>completed</td>\n",
       "      <td>func</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">return=1</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "9e4f313d-fe67-4a5c-a3fc-abb3ac40e9ca"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.run(workflow_path=\"workflow.py\", watch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c65df8e0",
   "metadata": {},
   "source": [
    "## View the results in the UI  Monitor Workflows tab\n",
    "\n",
    "<img src=\"../_static/images/parallel-workflow.png\" width=\"800\" >"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mlrun-base",
   "language": "python",
   "name": "conda-env-mlrun-base-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
